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Motivations and background 




Problem Presentation 




General-Purpose Operating Systems 

> Very effective for storing & managing multimedia contents 


> Designed for 


• average-case performance 


• serving applications on a best-effort basis 

>They are not the best candidate for serving real-time 
applications with tight timing constraints 

• like real-time multimedia 

• or computing with precise QoS assurance 
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Possible Solutions 





Overcoming limitations of a GPOS for multimediia 

> Large buffers used to compensate unpredictability 

• ==> poor real-time interactivity and no low-latency multimedia 

> One-application one-system paradigm 

• For example, for low-latency real-time audio processing (jack), 

gaming, CD/DVD burning, plant control, etc... 

> POSIX real-time extensions 

• Priority-based, no temporal isolation 

• Not appropriate for deploying the multitude of (soft) real-time 
applications populating the systems of tomorrow 

> Linux Real-Time Throttling extension 

• Designed for limiting, not guaranteeing 
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POSIX 

Real-TimeSched u I i n g 


Multi-queue priority-based scheduler 
Processes at same priority 

> Round-Robin (SCHED RR) 

> FIFO (SCHED FIFO) 

> Sporadic Server 
(see later) 


Process 





_ 

■ 

Max RT Prio 





Min RT Prio 






Non RT 
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Traditional RT Systems 

(and RriorltvSctieduii no) 


All deadlines respected as far as system behaves 
as foreseen at design time 

> Traditional (C, T) task model 

• C: Worst-Case Execution Time (WCET) 

• T: Minimum inter-arrival period 

Admission Control, e.g., for RM: 


i= 1 1 


C, 




i =1 


High priority 
( 2 , 6 ) 






t 


- 83 . 3 % 

Overall 

Load 


Low priority 

( 4 , 8 ) 


t 















Problems of 

BoodiM Soile d uln a 







High-priority processes may indefinitely delay 
low-priority ones 

> Coherent with the typical real-time/embedded scenario 
• Higher-priority processes are more important (e.g., safety critical) 
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Problems of 

Eri odiM Soile d uln a 


High-priority proesssss may indefinitely delay 
low-priority ones 

> Coherent with the typical real-time/embedded scenario 
• Higher-priority processes are more important (e.g., safety critical) 

>What if processes have same importance/criticality ? 


badjob 
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( 4 , 8 ) 
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Recently Proposed 
BgaJrijime ScJaeduJer(s) 



Features (schedulers implement) 

> Temporal isolation among tasks and task groups 

> Need for provisioning of reservation parameters 

(sporadic real-time task model) 

• runtime every period 

• Optional allowance to use more CPU if available (soft reservations) 

> Simple admission control scheme 

• May be disabled if custom user-space policy needed 

• Optional over-subscription possibility 

with graceful, controlled management of overloads 

> Priority-based, Deadline-based, mixed scheduling 

> Hierarchical scheduling 

• Attach more tasks as a whole to a single reservation 

• Nesting of groups and subgroups at arbitrary levels 
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AQuoSA EDF-based scheduler -f n 

EDF RT Throttling (a.k.a., The JRMOS Scheduler) 

> Parameters: runtime, period, cpu mask, tasks g 

Interactive 

• RT priorities of real-time tasks 

> cgroup-based interface 

• Problem of atomic changes to scheduling parameters 


SCHED.SPORADIC 

> Parameters: runtime, period, priority, low-priority 


> POSIX standard system call: sched_setscheduler() 

• Breaks binary interface & compatibility 

> Alternative system call: sched_setscheduler_ex() 


SCHED DEADLINE 

> Parameters: runtime, period, flags 

> system call: sched_setscheduler_ex() 



EVIDEnCE 
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Programming Paradigm 



SCHED DEADLINE 


struct sched_param_ex sp = { 

.sched_runtime = runtime_ts; // struct timespec 
.sched_deadline = deadline_ts; // struct timespec 
.flags = 0; 

}; 

sched_setscheduler_ex(pid, SCHED_DEADLINE, &sp); 

/* Now you get runtime_ts every deadline_ts guaranteed 7 





Adaptivity & Control of 
Resources in Embedded Systems 
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B P rogramming Paradigm 

IRMOS Scheduler 



Pre-requisite at run-time: mount cgroups 

> mkdir/cg 

> mount -t cgroup -o cpu,cpuacct cgroup /eg 

Reduce runtime for root-level tasks 

> echo 200000 > /cg/cpu.rt_rt_task_runtime_us 
(root-group runtime remains at default of 950000) 

Create group, with reservation of 10ms every 100ms 

> mkdir/cg/gl 

> echo 100000 > /cg/gl/cpu.rt_period_us 

> echo 10000 > /cg/gl/cpu.rt_runtime_us 

> echo 100000 > /cg/gl/cpu.rt_task_period_us 

> echo 10000 > /cg/gl/cpu.rt_task_runtime_us 

Attach task with tid=1421 


> echo 1421 > /cg/g 1/tasks 

Detach task 

> echo 1421 > /cg/tasks 

Attach process with pid=1700 

> fortid in 'Is/proc/1700/task'; do echo $tid > /cg/g 1/tasks; done 

Destroy group 

> rmdir/cg/gl 


IRMOS 

Interactive Realtime Multimedia Applications 
on Service Oriented Infrastructures 







Programming Paradigm 

IRMOS Scheduler with AQuoSA API 



AQuoSA 

qres_params_t p = (qres_params_t) { 
.Q = 10000, 

.Q_min = 10000, 

.P = 40000, 



.flags = 0 

}; 

if (qres_create_server(&params, &sid) == QOS_OK) { 
qres_attach_task(sid, 0, 0); 

/* Now we get 10ms every 40ms guaranteed 7 



Tommaso Cucinotta - Real Time Systems Laboratory (ReTiS) - Scuola Superipre Sant’Anna 


13 






Using resource reservations 
(and deadline-based scheduling) 
in the Jack low-latency infrastructure 




We modified Jack so as to 

> Use a deadline-based real-time scheduling policy 

> With automatic tuning of scheduling parameters 

• Period computed on the basis of the cycle duration/deadline 

• Budget computed through a feedback-based loop 

> With minimum changes to the Jack daemon and library 

> Without any change required to applications/clients 

We measured the obtained performance 

> No performance drop when running alone 

> Performance is kept despite other real-time workload 
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Jack Periodic Cycle 



Arbitrary complex DAG of computations 


Input 

driver 




Output 

driver 
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Jack Periodic CvcJe 



Arbitrary complex DAG of computations 


A91 computations must complete within the cycle 


Input 

driver 




Output 

driver 
















Jack Periodic CvcJe 



Arbitrary complex DAG of computations 

All computations must complete within the cycle 

Each computation belongs to a different process 



Output 

driver 
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Jack Periodic Cycle 



All client threads attached to a single reservation 



Output 

driver 
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Jack Periodic Cycle 



AID client threads attached to a single reservation 
Budget identified by feedback-based scheduling 
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Percentile estimator 

>Can be configured to estimate a percentile (can be 100%) 
of the observed consumed budget distribution over a 
moving window 

Additional] heuristics 

> Addition of a safety threshold (over-provision) 

> Temporary budget boost on new client 

> Temporary budget boost on xrun 

• (prevents further xrun from occurring after an xrun) 
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StiUU we couJd see some xruns 

> Some workload peaks cannot be foreseen 


e.g., MPEG decoding or MIDI synthesizer 


Solution 



> Use soft resource reservations 


• Tasks are allowed to run beyond budget exhaustion 

• The budget is still a minimum guarantee for the tasks 

> We used the SHRUB algorithm 

• Fair redistribution of unused bandwidth to active RT tasks 
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Experimental Results 





Testbed set-up 


Hardware 

> Processor: Intel E8400 @ 3GHz 

> Sound Card: Terratec EWX24/96 PCI 

Software 

> Linux Kernel: 2.6.29 

> RT Scheduler: POSIX and AQuoSA scheduler 

> Workload: jack and rt-app 

> rt-app parameters: 5ms every 40ms 



Tommaso Cucinotta - Real Time Systems Laboratory (ReTiS) - Scuola Superipre Sant’Anna 


24 








Jack @ 1333 us 
96 klj^. 128 samples 
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Concluding Remarks 


Summarizing 

> We tackled a challenging case-study for using 
resource reservations in Linux 



>We modified Jack to use a deadline scheduler 

>The critical issue was budget identification 

> The performance of Jack alone doesn't get worse 

>The set-up and deployment of a complex mix of real-time 
applications is simplified 

• Each one declares its own timing requirements 
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This work is far from being finished 

> Better handling of budget boost for new clients 

> Collaboration from clients with very dynamic workloads 

• e.g., MIDI synthesizer 

> Use a more recent scheduler 

> Experiment with the PREEMPT RT version of the deadline 
scheduler 

> Experiment with SMP and parallelization 


Tommaso Cucinotta - Real Time Systems Laboratory (ReTiS) - Scuola Superipre Sant’Anna 


28 







Related Publications 



> Hierarchical Multiprocessor CPU Reservations for the Linux 
Kernel 

F. Checconi, T. Cucinotta, D. Faggioli, G. Lipari 
OSPERT 2009, Dublin, Ireland, June 2009 


> An EDF Scheduling class for the Linux kernel 
D. Faggioli, F. Checconi, M. Trimarchi, C. Scordino 
RTLWS 2009, Dresden, October 2009 


> Access Control for Adaptive Reservations on Multi-User Systems 
T. Cucinotta 

RTAS 2008, St. Louis, MO, United States, April 2008 

> Self-tuning Schedulers for Legacy Real-Time Applications 
T. Cucinotta, F. Checconi, L. Abeni, L. Palopoli 
EuroSys 2010, Paris, April 2010 

> Respecting temporal constraints in virtualised services 
T. Cucinotta, G. Anastasi, L. Abeni 

RTSOAA 2009, Seattle, Washington, July 2009 


Tommaso Cucinotta - Real Time Systems Laboratory (ReTiS) - Scuola Superiore Sant’Anna 







Thanks for your attention 



Questions? 


http://retis.sssup.9t/people/tommaso 
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Deadline-based Scheduling 
for Temporal Isolation 

in Linux 





Deadline-based Scheduling 



Optimum for single-processor systems 

> Necessary and sufficient admission control test 

for simple task model: » c 

YY < 1 


Same problems of PS 

> Deadlines respected as far as the WCETs are respected 

> Things may go bad when 

• One or more tasks exhibit higher computation times than foreseen 

• One or more tasks behaves differently than foreseen 

- e.g., it blocks on a critical section for more than foreseen 

>The task that suffers may not be the misbehaving one 

Cannot provide Temporal Isolation unless... 
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Beaktime theory 



Reservation-based scheduling: (Q., P.) 

>“Q. time units guaranteed on a CPU every P. time units” 


A A A 
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> Independently of how others behave 

(temporal isolation) 
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Temporal Isolation 




Enforcement of temporal Isolation 

> Not only EDF scheduling 
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Temporal Isolation 





Enforcement of temporal Isolation 

>Once budget exhausted, delay to next activation period 


(5,9) 
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Temporal Isolation 



Is needed despite blocks/unblocks 

> Not only EDF scheduling 

block unblock 
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Temporal Isolation 




Is needed despite blocks/unblocks 

> Not only EDF scheduling 

block unblock 

▲ A A A 


(5,9) 
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See the “unblock rule” of the Constant Bandwidth 
Server (CBS, Abeni ’98) 


















SCHED SS 


> Provides a form of temporal isolation 

> Parameters: (Q, P, RT Priority, Low RT Priority) 

> Budget exhausted => lower the priority till next recharge 

> For every time interval in which the task executes, post a 
recharge of budget equal to the consumed CPU time one 

period apart .2 *2 A 1 A 1 1 A 


( 2 , 6 ) 


I~1 
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current 

budget 
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SCHED_SS may be analysed using FP techniques 

> Patching the standard for getting rid of the “bug” 


















IjRMOSBT Soiled ujer 

Des i a nGoals 




Replace real-time throttling 
Tight integration in Linux kernel 

Modification to the Linux RT scheduler 

neuse m many Linux Matures as 
possible 

Management of task hierarchies and 
scheduling parameters via cgroups 

>POSIX compatibility and API 

Efficient for SMP 

independent runqueues 








IRMOS Scheduler 



Slice the available computing power into 
reservations 




(Q i, Pi) 


CPU 3 
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hlierarclii.caJkSclieduJina 



Needed operations 

> create & destroy reservations 

> attach & detach tasks reservations 

> list tasks attached to reservations (and list reservations) 

> Standard operations: get & set parameters 
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Warning: features & parameters may easily grow 

> Addition of parameters, such as 

• deadline 

• desired vs guaranteed runtime (for adaptive reservations) 

> Set of flags for controlling variations on behaviour 

• work conserving vs non-conserving reservations 

• what happens at fork() time 

• what happens on tasks death (automatic reclamation) 

• notifications from kernel (e.g., runtime exhaustion) 

> Controlled access to RT scheduling by unprivileged 

applications (e.g., per-user “quotas”) 

> Monitoring (e.g., residual runtime, available bandwidth) 

> Integration/interaction with power management 
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